To do specific things to anger someone (less commonly, sexually arouse or otherwise elicit a strong reaction), especially intentionally or maliciously.
You really knew how to push my buttons.